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DETAILED ACTION 

Response to Amendment 

1 . The amendment filed 4/24/2008 is objected to under 35 U.S.C. 1 32(a) because it 
introduces new matter into the disclosure. 35 U.S.C. 132(a) states that no amendment 
shall introduce new matter into the disclosure of the invention. The added material 
which is not supported by the original disclosure is as follows. 

Regarding claim 1, the limiting phrase "and to communicate to the robot 
positions and orientations specified by the first specifying unit and the second specifying 
unit" is not found in the specification. Specifically it is not found that position and 
orientation of the second specifying unit is communicated to the robot. 

Applicant is required to cancel the new matter in the reply to this Office Action. 

Response to Arguments 

2. The Examiner thanks the Applicant for the timely reply received on 4/24/2008. 
Applicant's arguments with respect to claims 1 ,1 5,28 and their dependents have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Objections 

3. Claims 15,18,24,28 objected to because of the following informalities. 
Appropriate correction is required. 

Regarding claims 15,28, it appears that it was intended to insert a comma after 
the phrase "displaying a view comprising the composite augmented reality image". 
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Regarding claim 18, no antecedent basis is found for the phrase "the movable 
device". Claim 18 does not depend from claim 16, however a movable device is 
introduced in claim 16. 

Regarding claim 24, it depends from claim 22, then 15, wherein there is no 
antecedent basis found for "the local display device." Claim 24 does not depend from 
claim 23, however a local display device is defined in claim 23. 

Claim Rejections - 35 USC § 103 

4. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

5. Claims 1-7,9-20,22-28,30,33 rejected under 35 U.S.C. 103(a) as being 
unpatentable over 

i. Zamorano (US 2003/0179308 A1 ) in view of 

ii. Ebersole (US 2002/001 0734 A1 ) 

iii. Rahim (US 5,155,683 A). 

Regarding claim 1, Zamorano discloses a system for remote programming of an 
industrial robot (causing movement of the robot is a type of programming), the system 
comprising: a camera for capturing an image 121,192, the camera being movably 
located on the robot 125 at a local site, a registering unit 160,180 configured to 
generate graphics 191 and register the graphics generated by the first registering unit 
on the image from the camera, to provide a composite augmented reality image (fig. 1, 



Application/Control Number: 10/743,315 Page 4 

Art Unit: 2622 

H[0018, 0021-0023]), a remote display device 190 located at a remote site, for 
displaying a view comprising the composite augmented reality image (^[0020]), 

Zamorano does not expressly disclose a remote site physically separated from 
the local site, a first specifying unit configured to specify a position and an orientation in 
the remote site by determining a position and an orientation of the remote display 
device in relation to a fixed remote coordinate system wherein a position and orientation 
of the camera is dependent on the position and orientation specified by the first 
specifying unit, and wherein the first registering unit is adapted to register the generated 
graphics to the image in dependence on the position and orientation specified by the 
first specifying unit, a second specifying unit configure to specify a position and an 
orientation at the local site, a second registering unit configured to generate graphics 
and register the generated graphics on an environment at the local site or an image of 
the environment of the local site, in dependence on the position and the orientation 
specified by the second specifying unit, a local display device configured to display the 
environment at the local site and the graphics generated by the second registering unit 
projected on the environment, and a communication link configured to communicate 
information between the local site and the remote site, and to communicate to the robot 
positions and orientations specified by the first specifying unit and the second specifying 
unit, wherein the graphics generated by the first registering unit and second registering 
unit comprise information regarding movement of the robot. 

Ebersole discloses an inter-networked augmented reality (AR) system broadly 
defined as any combination of local AR/non-AR with remote AR/non-AR (Abstract, fig. 
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2) wherein the system comprises a remote site physically separated from the local site 
0|[0025]), a first specifying unit 33 configured to specify a position and an orientation in 
the remote site by determining a position and an orientation of the remote display 
device in relation to a fixed remote coordinate system (H[0025] - fixed remote 
coordinate system being the coordinate system utilized by a GPS system) wherein a 
position and orientation of the camera 34d is dependent on the position and orientation 
specified by the first specifying unit (fig. 1 1 , U[0058] - the tracking system data controls 
the viewing angle), and wherein the first registering unit 32 is adapted to register the 
generated graphics to the image in dependence on the position and orientation 
specified by the first specifying unit (fig. 1 1 , 1f[0058]), a second specifying unit 33 
configured to specify a position and an orientation at the local site (fig. 2, 3, ^[0029]), a 
second registering unit 32 configured to generate graphics and register the generated 
graphics on an environment at the local site or an image of the environment of the local 
site, in dependence on the position and the orientation specified by the second 
specifying unit fl|[0034]), a local display device 32 configured to display the environment 
at the local site and the graphics generated by the second registering unit projected on 
the environment (fig. 3, 1J[0028]), and a communication link 2 configured to 
communicate information between the local site and the remote site, and to 
communicate to the robot positions and orientations specified by the first specifying unit 
and the second specifying unit (U[0025, 0030, 0032, 0042-0047]). Therefore it would 
have been obvious to one having ordinary skill in the art at the time of the invention to 
modify the robot camera taught by Zamorano to further allow the AR display device to 
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be remotely located and head mounted, the robot camera be remotely controlled by the 
tracking of the remote head mounted display, to allow a second local AR head mounted 
display device, and to allow a communication link to inter-link all components in order to 
further enhance the AR system abilities and experience (Abstract, ^[0004]). 

Rahim discloses an augmented reality system comprising wherein the graphics 
generated by a registering unit comprise information 12,18 regarding movement of a 
camera robot vehicle V (Abstract, figs. 1-5, col. 8 lines 27-44). Therefore it would have 
been obvious to one having ordinary skill in the art at the time of the invention to modify 
Zamorano as viewed to further incorporate graphical information regarding movement of 
the robot as taught by Rahim in order to allow the viewer to review the projected path 
(col. 5 line 35). 

Regarding claim 2, Zamorano as viewed discloses the system according to claim 

1 , wherein said first specifying unit comprises a tracking unit adapted to determine a 
position and orientation of a movable device located at the remote site, the first 
registering unit adapted to register the generated graphics on the image in dependence 
of the position and orientation of the movable device, and the camera is arranged such 
that its position and orientation are dependent on the position and orientation of the 
movable device (Ebbersole fig. 3, 4, 1 1 , H[0058, 0059]). 

Regarding claim 3, Zamorano as viewed discloses the system according to claim 

2, wherein said movable device is the remote display device 45 (Ebbersole). 
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Regarding claim 4, Zamorano as viewed discloses the system according to claim 
1 , wherein the industrial robot is located at the local site, wherein the camera is 
mounted on the industrial robot (Zamorano fig. 1) and wherein the industrial robot is 
arranged in such a manner that movement of the industrial robot depends on the 
specified position and orientation (Ebbersole 1J[0058]). 

Regarding claim 5, Zamorano as viewed discloses The system according to 
claim 1, further comprising a graphical generator configured to generate a graphical 
representation, wherein the registering unit is adapted to generate graphics based on 
the graphical representation (Ebersole figs. 6-8, 10, page 2 ]|[0028]). 

Regarding claim 6, Zamorano as viewed discloses the system according to claim 
1 , further comprising operator input means 35 located at the remote site and configured 
to feed data related to the graphics to be displayed to the system, wherein the system is 
adapted to generate the graphics based on said data (Ebersole fig. 3, 4, 6, 1J[0039]). 

Regarding claim 7, Zamorano as viewed discloses the system according to claim 
6, wherein said operator input means comprises a pointing device and a tracking unit 
configured to determine a position of the pointing device and wherein the system is 
adapted to generate a graphical representation of a point pointed out by the pointing 
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member based on the position of the pointing device (Ebersole 1J[0031 , 0050] - 
graphical representation being an AR pointer). 

Regarding claim 9, Zamorano as viewed discloses the system according to claim 
1 , further comprising a second movable device 47 located at the local site, wherein the 
second specifying unit comprises a second tracking unit to determine the position and 
the orientation of the second movable device (Ebbersole figs. 2-4). 

Regarding claim 10, Zamorano as viewed discloses the system according to 
claim 9, wherein said second movable device is the local display device 47 (Ebbersole). 

Regarding claim 11, Zamorano as viewed discloses the system according claim 
9, further comprising a second camera 34a for capturing an image, the camera being 
arranged in a fix relation to the second movable device 45, and wherein the second 
registering unit is adapted to register the generated graphics generated by the second 
registering unit to the image from the second camera, to provide a composite 
augmented reality image, and wherein the local display device 45 is adapted to display 
a view comprising the composite augmented reality image (fig. 3, 4, ^[0030]). 

Regarding claim 12, Zamorano as viewed discloses the system according to 
claim 1 , wherein the remote display device is adapted to display a view seen from a first 
visual angle that depends on the position and orientation received from the first 
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specifying unit and wherein the local display device is adapted to display the same view 
as the remote display device seen from a second visual angle that depends on the 
position and orientation received from the second specifying unit (Ebbersole Title, 
Abstract, page 5 U[0057] - internetworked AR). 

Regarding claim 13, Zamorano as viewed discloses the system according to 
claim 1, further comprising means for transferring voices between the remote and the 
local site via the communication link (Ebbersole U[0025, 0060]). 

Regarding claim 14, Zamorano as viewed discloses the system according to 
claim 1, wherein the communication link comprises a network (Ebbersole fig. 1, 
10032]). 

Regarding claim 15, Zamorano as viewed (refer to the rejection of claim 1) 
discloses a method for remote programming of an industrial robot by remotely 
displaying an augmented reality view comprising graphical information overlaid an 
image captured at a local site, the method comprising: specifying a position and an 
orientation at a remote site that is physically separated from the local site with a tracking 
unit carried by or arranged on an operator at the remote site, positioning and orienting 
the robot such that a camera arranged on the robot assumes the specified position and 
orientation, obtaining an image from the camera, generating first graphics, generating a 
composite augmented reality image with a registering unit based on the image, the 
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generated first graphics, and the specified position and orientation, displaying a view 
comprising the composite augmented reality image, specifying a position and an 
orientation in the local site, displaying a second view comprising an environment of the 
local site and the generated first graphics projected on the environment in dependence 
of the locally specified position and orientation, and controlling movements of the robot 
at the local site and teaching the robot one or more waypoints to carry out a task 
(Rahim col. 7 lines 14-26 - teaching being instructed, task being a proposed 
movement). 

Regarding claim 16, Zamorano as viewed discloses the method according to 
claim 15, wherein specifying a position and an orientation comprises determining a 
position and an orientation of a movable device located at the remote site and wherein 
the camera is positioned and oriented according to the position and orientation of the 
movable device (Ebbersole fig. 3, 4, 1 1 , 1|[0058, 0059]). 

Regarding claim 17, Zamorano as viewed discloses the method according to 
claim 16, wherein said movable device comprises a remote display device and wherein 
said view comprising the composite augmented reality image is displayed on the remote 
display device (Ebbersole fig. 3, 10028]). 

Regarding claim 18, Zamorano as viewed discloses the method according to 
claim 15, wherein the camera is mounted on the robot (Zamorano fig. 1), the method 
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further comprising controlling movements of the robot according to the position and 
orientation of the movable device (Ebbersole 1f[0058]). 

Regarding claim 19, Zamorano as viewed discloses the method according to 
claim 15, further comprising obtaining data related to the generated first graphics to be 
displayed, and generating the first graphics based on said data (Ebbersole fig. 3, 4, 6, 
10039]). 

Regarding claim 20, Zamorano as viewed discloses the method according to 
claim 15, further comprising receiving information about the position of a pointing device 
and generating first graphics representing a point pointed out by the pointing member, 
based on the position of the pointing device fl[[0031 , 0050] - first graphics being an AR 
pointer). 

Regarding claim 22, Zamorano as viewed discloses the method according to 
claim 15, wherein specifying a position and an orientation in the local site comprises 
determining a position and an orientation of a second movable device located at the 
local site (Ebbersole figs. 2-4, page 2 ^[0029]). 

Regarding claim 23, Zamorano as viewed discloses the method according to 
claim 22, wherein the second movable device 45 comprises a local display device and 
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wherein said second view, comprising the environment of the local site and the 
graphics, is displayed on the local display device (Ebbersole figs. 2-4, 1J[0029]). 

Regarding claim 24, Zamorano as viewed discloses (refer to Ebbersole) the 
method according to claim 22, further comprising capturing an image from a second 
camera 34a being arranged in a fixed relation to the second movable device 45 (fig. 4), 
registering the generated graphics on the image from the second camera, to provide a 
composite augmented reality image, and displaying a view comprising the composite 
augmented reality image on the local display device 45 (fig. 3, 4, ^[0030]). 

Regarding claim 25, Zamorano as viewed discloses the method according to 
claim 15, further comprising generating second graphics and displaying the second view 
comprising the environment of the local site and the second graphics projected on the 
environment in dependence of the specified position and orientation (Ebbersole fig. 3, 
10028, 0034, 0040]). 

Regarding claim 26, Zamorano as viewed discloses (refer to Ebbersole) the 
method according to claim 25, further comprising generating a local graphical 
representation, generating a remote graphical representation, transferring the local and 
remote graphical representations between the local and the remote site (fig. 1 , page 3 
1J[0032]), generating the remote first graphics based on the local and the remote 
graphical representation, and generating the second graphics based on the local and 



Application/Control Number: 10/743,315 Page 13 

Art Unit: 2622 

the remote graphical representation (1f[0037, 0042-0047, 0058-0060] - the collaboration 
of textual messages involves local and remote graphical representations in the form of 
text wherein when displayed they become a part of the local and remote graphics). 

Regarding claim 27, Zamorano as viewed discloses the method according to 
claim 15, wherein the view displayed at the remote site comprises the environment of 
the local site and the overlaid graphics seen from an visual angle that depends on the 
position and orientation specified in the remote site and the view displayed in the local 
site comprises the environment of the local site and the overlaid graphics seen from an 
visual angle that depends on the position and orientation specified in the local site 
(Ebersole Title, Abstract, U[0057]). 

Regarding claim 28, Zamorano as viewed (refer to the rejection of claim 15) 
discloses a computer program product (Ebersole fig. 2, 3, U[0027], Appendix A), 
comprising: a computer readable medium; and computer program instructions recorded 
on the computer readable medium and executable by a processor for performing a 
method for remote programming of an industrial robot by remotely displaying an 
augmented reality view comprising graphical information overlaid an image captured at 
a local site, the method comprising specifying a position and an orientation at a remote 
site that is physically separated from the local site with a tracking unit carried by or 
arranged on an operator at the remote site, positioning and orienting a camera, located 
at the local site, according to the specified position and orientation, obtaining an image 
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from the camera, generating first graphics, generating a composite augmented reality 
image with a registering unit based on the image, the generated first graphics, and the 
specified position and orientation, displaying a view comprising the composite 
augmented reality image, specifying a position and an orientation in the local site, 
displaying a second view comprising an environment of the local site and the generated 
first graphics projected on the environment in dependence of the locally specified 
position and orientation, and controlling movements of the robot at the local site and 
teaching the robot one or more waypoints to carry out a task. 

Regarding claim 30, Zamorano as viewed discloses use of a system according 
to claim 1 for remote programming of an industrial robot by controlling movements of 
the robot at the local site and teaching the robot one or more waypoints to carry out a 
task (Rahim col. 7 lines 14-26 - teaching being instructed, task being a proposed 
movement). 

Regarding claim 33, Zamorano as viewed discloses use of the system according 
to claim 1 for a paint application (fig. 10, 10051-0053]). 

6. Claims 31-32 rejected under 35 U.S.C. 103(a) as being unpatentable over 

i. Zamorano (US 2003/0179308 A1 ) in view of 

ii. Ebersole (US 2002/001 0734 A1 ) 
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iii. Rahim (US 5,155,683 A) 

as applied to claim 1 1 above, and further in view of 

iv. Ellenby (US 5,815,411 A). 

Regarding claim 31, Zamorano as viewed does not expressly disclose the 
system according to claim 1 1 , further comprising a handheld display device comprising 
the display member and the camera. 

Ellenby discloses the system according to claim 11, further comprising a 
handheld display device (fig. 1) comprising the display member 13 and the camera 9 
(figs. 1-4 - binoculars being handheld). Therefore it would have been obvious to one of 
ordinary skill in the art at the time of the invention to allow the head mounted displays 
taught by Zamorano as viewed to be a handheld tracked display as taught by Ellenby in 
order to offer the user a preferred method of viewing (col. 3 line 63, col. 9 lines 47-51 ). 

Regarding claim 32, Ellenby discloses the system according to claim 31 , wherein 
the handheld display device is arranged so that the user seems to look directly through 
the display (col. 9 lines 47-51 , Claim 4). 

Conclusion 

7. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See M PEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

8. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Dalziel (US 5,579,444) teaches a programmed industrial robot arm with camera. 

Corby (US 5,706,195) teaches an augmented reality system. 

Milgram (July 1995) teaches remote robot programming with augmented reality. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to PAUL SAUNDERS whose telephone number is 
(571)270-3319. The examiner can normally be reached on Mon-Thur 9am-5pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, NgocYen Vu can be reached on 571 .272.7320. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/PS/ 

7/17/2008 



/Ngoc-Yen T. VU/ 
Supervisory Patent Examiner, Art Unit 2622 



